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Abstract 

This paper recalls the definition of consistency for pairwise comparison matrices and briefly 
presents the concept of inconsistency index in connection to other aspects of the theory of 
pairwise comparisons. By commenting on a recent contribution by Koczkodaj and Szwarc, it 
will be shown that the discussion on inconsistency indices is far from being over, and the ground 
is still fertile for debates. 
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1 Introduction 

In problems of choice, decision makers often face the selection of a single best alterna¬ 
tive from a feasible set. In modern methodologies for decision making, as for instance 
the Analytic Hierarchy Process (AHP) [28], the use of pairwise comparisons between 
alternatives has been strongly advocated. Comparing two alternatives at a time and ex¬ 
pressing the degree of preference of one to the other, is supposed to simplify the original 
problem by decomposing it into smaller and more easily tractable ones. 

The most widely known representation of valued preferences is based on the pair¬ 
wise comparison matrices used by Saaty in the AHP. Given a finite non-empty set of 
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alternatives, {0i,..., 0„}, a pairwise comparison matrix is a matrix A = {aij)nxn, with 
aijOji = 1 Vi, j, where Oij > 0 represents the degree of preference of 6i to 6j. Moreover, 
a pairwise comparison matrix is called consistent if the following condition holds, 

O'ik — CLijCLjk Vi , j,k. (1) 

Such a condition is the formalization of a desirable property of cardinal transitivity. 
Namely, the value of each direct comparison Oik is precisely backed up by all indirect 
comparisons aijOjk- For example, if 6i is considered twice as good as 9j which, in turn, is 
also considered twice as good as 0k, it appears reasonable that 6i be four times as good 
as 6k- Furthermore, condition 0 can be restated in the following equivalent ways: 

• There exists a (priority) vector w = {wi,... ,Wn) such that a^- = Wijwj 'ii,j 

• A has unitary rank, i.e. rank(A) = 1 

• The Perron-Frobenius eigenvalue of A is equal to n, i.e. Amax = n. 

The scientific debate has centered on the methods used to quantify the deviation between 
the preferences of a decision maker and the condition of full consistency. Such quan¬ 
tification is usually done by functions which, in the literature, are called inconsistency 
indices. 

Mathematically speaking, an inconsistency index is a function I which maps pairwise 
comparison matrices into the real line, so that /(A) G M is an estimation of the in¬ 
consistency of A. Usually, the greater /(A), the higher the level of inconsistency. For 
some inconsistency indices, threshold values r have been proposed, so that, if /(A) > r, 
then the matrix A is considered too inconsistent and the decision maker has to re¬ 
vise his judgments. It is worth commenting that there is not a meeting of minds on 
what the thresholds should be and how they could be estimated. For example, Saaty 
proposed r = 0.1 for his index CR, where 0.1 means that all the matrices with inconsis¬ 
tency smaller than the 10% of the inconsistency of a random matrix should be accepted. 
Koczkodaj, instead, proposed r = 1/3 for the index K. Interestingly, many indices have 
been introduced in the literature without studying their thresholds. 

A large number of indices have been proposed; ten of them were numerically analyzed 
[5], and some more were proposed more recently [25]. Some indices were even proven 
proportional [6] or equivalent |26j to each other, but what is more remarkable is how 
differently they may behave. Figure[^shows an example where two inconsistency indices 
are compared via numerical simulations: each point in the scatter-plot corresponds to a 
pairwise comparison matrix and its coordinates on the axes represent its inconsistency 
evaluated by the two inconsistency indices proposed by Barzilai [3| and Gass and Rapcsak 
respectively. Two significant cases are highlighted with a larger and clearer dot. 
In both cases an index evaluates a matrix as extremely inconsistent, whereas the other 
one indicates that the same matrix is almost consistent. Such discordant behavior is 
not limited to the two indices presented in the example. As different indices often yield 
different results and conclusions, choosing among them really makes a difference. All 
this just adds up to the need of a deeper understanding of inconsistency indices. 
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Figure 1: Indices IM m and RE [3] computed on 2,000 pairwise comparison matrices 
of order 6. 


In the next section, the paper will first recall the central role of inconsistency indices 
in the theory of pairwise comparison matrices. Thereafter, it will discuss the recent 
contribution by Koczkodaj and Szwarc [Mj, which presented a critical assessment of a 
number of inconsistency indices implying their unsoundness as inconsistency estimators. 
By challenging their conclusions, this paper will hopefully put forward a more tolerant 
viewpoint. 


2 The central role of inconsistency indices 

The role played by inconsistency indices has not been limited to the mere estimation of 
the irrationality of pairwise comparisons. With its ramifications, the concept of consis¬ 
tency, and of its counterpart inconsistency, has permeated the whole theory of pairwise 
comparisons. 

A fundamental step in the exploitation of the information contained in a pairwise 
comparison matrix is the synthesis of this latter into a priority vector w = (rci,..., Wn)- 
Many of the methods used to estimate w are mathematically connected to an index 
of inconsistency. For example, the eigenvector method is associated with Amax and 
consequently to Cl [28]. Another example is the Geometric Consistency Index [T|, 
which is connected to the so-called Geometric Mean method used to determine w ra¬ 
in addition, many methods to estimate w seek for the consistent matrix {w*/w*)nxn 
which is the closest to A. In the literature, the distance between {w*/w*)nxn and A has 
often been interpreted as an estimation of the inconsistency of A. 

When the number of alternatives grows, it can happen that a decision maker is unable 
to express all the n(n —1)/2 independent pairwise comparisons necessary to complete the 
matrix. Many methods are concerned with the optimal completion of matrices. That is, 
given a subset of comparisons, how can we estimate the missing ones? One very common 
approach is that of minimizing an inconsistency index. In this case the missing values are 
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the variables. For example, given the following incomplete pairwise comparison matrix, 


/ 1 

2 

013 

014^ 

1/2 

1 

1/3 

1 

l/ai3 

3 

1 

2 

\l/ai4 

1 

1/2 

1 / 


one might want to solve the optimization problem 

minimize H-^) 
subject to oi 3 ,ai 4 > 0 


( 2 ) 


(3) 


to find the most suitable values of ois and 014 with respect to a given inconsistency index 
I. Bozoki et al. [1], Shiraishi et al. [29], Koczkodaj |23) and Lamata and Pelaez m 
all proposed optimization problems in the form of Q. The importance of incomplete 
preferences has been of ever increasing interest in the literature, as also supported by 
the recent surveys by Ishizaka and Labib m and Ureha et al. m- 

Optimization problems have also been used recently to improve the consistency of 
preferences when these latter ones are excessively inconsistent. Examples are the opti¬ 
mization problems proposed by Dong and Herrera-Viedma m and Dong et al. m to 
mitigate the inconsistency which arises, for example, due to the due to the nature of 
Saaty’s scale. 

There exists alternative representations of valued preferences where the pairwise com¬ 
parisons are equivalently expressed on other representation domains. The foremost are 
(i) reciprocal relations m, often called fuzzy preference relations in the fuzzy sets 
literature m, and (ii) additive preference relations, loosely related with Fishburn’s 
skew-symmetric additive representation of preferences [16]. For these representations 
of preferences, inconsistency indices have also been proposed. One can consider, for 
instance, those introduced by Barzilai [3], Ji and Jiang [20] and Yuen [32] for additive 
preferences. Given the existence of transformations between different representations 
[ 8 ], results obtained in one context might be directly applicable in another. 

Inconsistency indices have also been considered in conjunction with group decisions 
and consensus reaching, e.g. by Chiclana et al. [9] and Dong et al. |14j . This seems to 
support the thesis that aspects of pairwise comparisons which were studied separately 
are, conversely, intertwined. 

Especially in light of the most recent development, it should be clear that inconsistency 
indices play a central role in the theory of valued preference relations. It naturally follows 
that a fair and rigorous analysis is just necessary. 


3 (Re)opening the debate on inconsistency indices 

In spite of the variety of indices proposed in the literature, until recent times there 
were no axiomatic studies. Two recent proposals of axiomatization were by Brunelli and 
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Fedrizzi 00 and by Koczkodaj and Szwarc [M]- Besides presenting separate axiomatic 
systems, the two proposals differ in the conclusions, with Koczkodaj and Szwarc being 
more drastic. For sake of brevity, the reader’s familiarity with Koczkodaj and Szwarc’s 
results is assumed, even though the main concepts will be recalled. 

In their paper, Koczkodaj and Szwarc proposed a set of three axioms. The first two 
axioms are simple regularity conditions while the third seems to be more constraining. 
These three axioms are in the form of reasonable properties and serve to implicitly define 
inconsistency indices. In line with this, the authors claimed that their axiomatic system 
“allows us to define proper inconsistency indicators”. Therefore, one might suppose 
that if a function satishes their axioms, then they should grant it the status of “proper” 
inconsistency index. 

3.1 The triad and the matrix 

The interpretation of the third axiom of Koczkodaj and Szwarc is problematic, because 
of its ambiguous definition and the soundness of the adduced motivation. According to 
them 

“It is a reasonable expectation that the worsening of a triad, used in the 
definition of consistency, cannot make the entire matrix more consistent”. 

The following example challenges their expectation. Consider the matrix 


/1 

2 

2 

4 

7 \ 

1/2 

1 

4 

1 

3 

1/2 

1/4 

1 

1 

4 

1/4 

1 

1 

1 

2 

Vl /7 

1/3 

1/4 

1/2 

1/ 


The entry 015 appears in the triads (ai 2 ) 025 , ^ 15 )) (ois, ossj 015 ) and ( 014 , 045 , 015 ). Let 
us write them down more explicitly. 

(012,025,015) = ( 2 , 3 , 7 ) 

(ai3, 035,015) = ( 2 , 4 , 7 ) 

(014,045,015) = ( 4 , 2 , 7 ) 

Now, if we slightly increase the value of 015 from, say, 7 to 7 . 5 , this worsens the local 
inconsistency of the triad (012, 025, 015). By following the assumption by Koczkodaj and 
Szwarc, this should not imply an enhancement of the consistency of the entire matrix. 
It occurs that this assumption neglects the incidence of 015 on the inconsistency of other 
triads. In fact, in this case, both (013,035,015) and (014,045,015) benefit from such a 


^Preliminary results on this set of axioms were already presented as “M. Brunelli and M. Fedrizzi, 
Characterizing properties for inconsistency indices in the analytic hierarchy process, Abstracts of 
MCDM 2011”, and “M. Brunelli and M. Fedrizzi, Characterizing properties for inconsistency indices 
in the AHP, Proceedings of ISAHP 201V' 
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change and become less inconsistent. The example with a matrix of order 5 was purely 
illustrative, and drawing from basic graph theory we know the following proposition. 

Proposition 1. Given a pairwise comparison matrix A of order n and its triples 
{aij,ajk,aik) VI < z < j < fc < n, then a single comparison aij {i ^ j) appears in 
exactly (n — 2) triples. 

Furthermore, one could easily build up an example of matrix of order n where, by 
changing one entry and its reciprocal we worsen one single triad but we ameliorate all 
other (n — 3) of them. Hence, in light of the fact that, except for matrices of order 
3, changing one entry has effects on a plurality of triads, the assumption proposed by 
Koczkodaj and Szwarc becomes unattainable and therefore useless. 

3.2 The role of an axiomatic system 

As correctly claimed by Koczkodaj and Szwarc [21], in this context, an axiomatic system 
serves to define a set of functions which are, in principle, all suitable to represent a given 
characteristic of a mathematical object. Many other axiomatic systems work in a similar 
way as, for instances, the axiomatic definition of distance in metric spaces or of norm in 
linear spaces. It should follow that, if a function I satisfies their three axioms, then this 
same function ought to be respected as an inconsistency index (at the very least by the 
authors who proposed the axioms). 

In spite of the ambiguous formulation (especially concerning the third axiom), by con¬ 
sidering a 3 X 3 pairwise comparison matrix 

/ 1 Uij 

Asxs = I l/oij 1 

yl/fljfc Xjaj]^ 

the axiomatic system by Koczkodaj and Szwarc can 

Axiom 1. //Asxs is consistent, then /(Aaxs) = 0. 

Axiom 2. /(Asxs) G [0,1[. 

Axiom 3. /(Asxs) is a quasi-convex function with respect to Oij, Ojk, Oik > 0, with global 
minimum attained when aik = aijOjk- 

In their research, Koczkodaj and Szwarc concluded that the index 

CI(A) = 

n — 1 

is inadequate to capture inconsistency, but they do not mention whether it satisfies the 
axioms or not. However, by examining the literature one finds that the behavior of 
Cl with respect to the entries of A was already studied by others. Bozoki et al. |1] 
examined some properties of generalized convexity of C/ and referred to previous results 


^ik\ 

Oijk I ) 

1 / 

be formalized as follows. 
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by Kingman m and Aupetit and Genest [2] . Considering a more relaxed version of the 
second axiom by Koczkodaj and Szwarc, the following is a direct corollary of the above 
mentioned studies. 

Corollary 1. If we substitute Axiom 2 with the more relaxed requirement that /(Asxs) G 
[0,oo[, then index Cl satisfies all the properties proposed by Koezkodaj and Szwarc. 

For an illustrative example, consider the matrix 

/I 3 X \ 

A = 1/3 1 1/2 . 

\l/x 2 1 / 

Figure [^illustrates the quasi-convex behavior of Cl (A) as a function of x. For the proofs 

C/(A) 



Figure 2: Quasi-convexity of C/(A) with respect to x. 

that several other inconsistency indices satisfy the axiomatic framework by Koczkodaj 
and Szwark, the reader can refer to the axiomatic study by Brunelli and Fedrizzi [7]. 
Moreover, one shall note that the index Cl could be normalized, e.g. by dividing its 
value by its upper bound as specified by Aupetit and Genest [2], and thus, for practical 
purposes, it would satisfy all the axioms by Koczkodaj and Szwarc. In conclusion, it 
appears that Cl and many other indices satisfy the axioms proposed by Koczkodaj 
and Szwarc and, by using their own words, ought to be considered proper inconsistency 
indices. 

3.3 One rotten apple does not necessarily spoil the barrel 

To support their criticisms against Cl, and incidentally towards a number of other in¬ 
consistency indices, Koczkodaj and Szwarc considered the following pairwise comparison 
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matrix, where the order n is not fixed, 


^Ks{x) 


1 1 


\l/x 1 


1 x\ 
1 1 




( 4 ) 


Their main result is that, it does not matter the value of x > 0, when n grows to infinity, 
CI{Aks{x)) tends to zero. The authors considered this a reductio ad absurdum and used 
it to invalidate Cl and all indices sharing the same property. Seemingly, Koczkodaj and 
Szwarc must have thought that the absurdity was self evident, since they did not further 
motivate it. 

Let us try to interpret the result in a semi-real context, just with the restrictions imposed 
by reasoning in a limiting sense. Especially when n is large, it is very plausible that the 
judgment ain = x is the outlier, since it is not in accordance with all the others. Hence, 
din = X can be seen as an error made by the decision maker in expressing opinions. Let 
us now propose the short story of Laura and her awful grade. Its resemblance with the 
problem at stake will hopefully be clear to the reader. 

Suppose that Laura is a student in a school where it is possible to get an 
arbitrarily bad grade. Further suppose that one day Laura gets a dreadful 
grade, say x. After this unfortunate episode, Laura keeps on getting magnif¬ 
icent grades in the exams on the same subject. Is it fair to allow her, at some 
point in the future (assume that she has infinitely much time), to be able to 
make up for the initial bad grade and eventually be considered a sufficiently 
good student? 

This is definitely a moot point. Contrasting opinions on the issue might exist, and the 
mere existence of different, yet respectable opinions, on a proposal seem to clash with 
the very same proposal being declared absurd. One simple argument in favor of Laura 
is that, if a slightly bad grade can be made up for, then also worse grade can be made 
up for, just in a longer time. And if not, where would one draw the (arbitrary) line 
between repairable and unrepairable bad grades? Furthermore, it is noteworthy that 
a school using arithmetic mean—which is often the case—allows compensations among 
grades and Laura would be allowed to repair the damage made by the initial bad grade. 

Likewise, when applied to the case of inconsistency of preferences, the story of Laura 
shows us that the absurdum is not so absurd, and that approaches contemplating the pos¬ 
sibility of compensating a local inconsistency by means of many consistent comparisons 
ought to be respected. 

Moreover, on a more computational note, the relative incidence of a wrong judgment, 
or of a relatively small number of judgments, decreases with the growth of the order 
of the matrix. In fact, the relative importance of few outliers in processes such as the 
determination of the priority vector diminishes as the order of the matrix grows. Indeed, 
this type of argument is borrowed from statistical reasoning, where it is accepted that 



the incidence of noisy observations can be mitigated by the growth of the sample size. A 
pairwise comparison matrix is not a completely different problem since more comparisons 
are required than those necessary to elicit a weight vector. Besides, many methods for 
determining the weight vector are borrowed from statistics, as for instance the Least 
Squares and the Logarithmic Least Squares which, in a way, consider the entries of A 
as the sample from which w is estimated. 


3.4 On the index K 


Koczkodaj and Szwarc supported the use of the following index mm 


K{A) 


max < min 
i<j<k [ 


Ojj k 
^ik 


1 - 


^ik 


^jk 


(5) 


on the ground that the alleged reductio ad adsurdum does not apply to it. Hereafter, 
we shall see that even index K, in some particular cases, gives rise to doubts on its ca¬ 
pacity to capture inconsistency. This can be shown starting with the following example. 
Considering the matrix Aks{x) in Q and fixing, for example, x = 2 we obtain. 


Ai^5(2) 


/ 1 1 
1 1 


1 1 

Vl/2 1 


1 2 \ 
1 1 


1 1 
1 1/ 


( 6 ) 


With the currently proposed acceptance threshold set at iL < 1/3 [H], one computes 
iL(Axs(2)) = 1/2 for all n, which means that Aks{‘^) is not sufficiently consistent and 
thus needs revision. Besides, the peculiar feature of K is its insensitivity to n. In fact, 
it is trivial to prove that, in the case of Aii'5(2), K is invariant with respect to n. Seen 
from this point of view, and considering the pairwise comparison matrix in Q it seems 
more realistic that the value of inconsistency of Aks{‘^) varies with n. 

Let us consider the following two matrices, to show that the superiority of K ought 
not to be taken for granted. 


Ai = 


^ 2.001 


1 1 
1 1 
1 1 
1 1 
1 1 


1 2 . 001 \ 
1 1 

1 1 

1 1 

1 1 / 


A 2 


/I 1 2 1 2\ 

1 1 12 1 

1/2 1 112 
11/2111 
\l/2 1 1/2 1 1/ 


Although K{Ai) ^ K(^A. 2 ^^ it would also be legitimate to expect an index I to yield 
the opposite, L(Ai) < /(A 2 ). The situation becomes more evident if we consider all the 
possible triads {aij,ajk,aik) with i < j < k. In the case of Ai only 3 triads out of 10 
are inconsistent and all the others appear perfectly consistent. Conversely, in A 2 all the 
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triads are inconsistent. 

The findings are even more evident if we reckon that this example can be generalized 
with respect to the order of the matrix and the values of the non-unitary entries. 

1 1 a 1 a 

1 1 1 a 1 

1 /a 1 11 a 

1 1 /a 1 1 1 


For e > 0, it is K{A 3 ) > K{A 4 ^). Note moreover that, if we consider an arbitrarily 
small e > 0 , we have a ~ a + e, and the differences in the violations of the transitivities 
become negligible. Therefore, since the extents of violations are approximately equal, 
it makes more sense to count the number of violations instead of their intensities. In 
A 3 the number of triads containing inconsistencies is equal to (n — 2) while the total 
number of triads is ( 3 ). Their ratio, simplified, is 

number of inconsistent triads n — 2 6 

number of triads Q) (n — l}n 

This ratio, and with it the relative importance of the inconsistent triads with respect to 
the whole, tends to zero as n progresses to infinite. Nonetheless, by considering K we 
have that the inconsistency of A 3 is always greater than the inconsistency of A 4 (where 
all the triads are intransitive!). 

By writing (/(a^, a^fc) = min{|l - aik/{aijajk)\,\'^ - {aijajk)/aik\} we can rewrite 

K{A) = max {(p{aij,ajk,aik)} 
i<j<k 

and thanks to the properties of max we can enunciate the following simple proposition. 

Proposition 2. Index K is not a strictly monotone increasing function with respect to 
the (p{aij,ajk,aik), except for the greatest of them. 

In words, the peculiar behavior of K, illustrated above, is due to the fact that index 
K considers the most inconsistent triad only, whereas many other indices average the 
inconsistencies coming from different triads. 

4 Conclusions 

Let us momentarily forget about the subject matter and instead consider the following 
two norms of the vector x = (xi, ..., Xn) G K”, 

A 2 (x) = max \xi\. 

2=l,...,n 
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Ideally, as also proposed by Horn and Johnson [18], a norm should capture the “size” 
of a vector. The two norms A^i and N 2 capture two views on the “size” of x; namely, 
Ni averages the component-wise contributions whereas N 2 considers only the greatest 
one. It follows that they can give very different answers to the question concerning the 
“size” of the vector x. Nonetheless, they are both norms as they respect the axiomatic 
conditions for norms. 

Considering the vector x = (J, 0,0,..., 0,0) G M"', one can see that it does not matter 
how large J > 0 is, as we increase n, iVi(x) will always tend to zero. Still, nobody has 
ever used this argument to invalidate A^i, and both Ni and N 2 coexist under the same 
umbrella. It has been up to applied mathematicians, economists, computer scientists 
and others, to distinguish between different norms, be aware of their properties, and 
select the one which seems the most suitable for a given context or situation. 

It is straightforward to transpose this discussion to the realm of inconsistency indices. 
If some researchers propose a sound axiomatic system in which an inconsistency index 
fits, then this inconsistency index should not be considered incorrect, in the same way 
as neither Ni nor N 2 should be classified as an incorrect norm. 

Different axiomatic systems for inconsistency indices have been proposed. They can 
be discussed and hopefully improved. Within these systems there exists a wealth and a 
diversity of indices which should be seen as a richness, and not as a threat. This paper 
showed examples where the behavior of index K was questionable. Shall we therefore 
label it as an unsound inconsistency index? The answer of the author of this paper 
is negative. Different indices have different strong points and weaknesses, but each of 
them represents a valuable point of view on the same phenomenon: inconsistency of 
preferences. 
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